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Foreword 



This Technical Specification (TS) has been produced by ETSI Technical Committee Speech Processing, Transmission 
and Quality Aspects (STQ). 

The present document is part 3 of a multi-part deliverable covering the QoS aspects for popular services in GSM and 
3G networks, as identified below: 

Part 1 : "Identification of Quality of Service aspects" ; 

Part 2: "Definition of Quality of Service parameters and their computation"; 

Part 3: "Typical procedures for Quality of Service measurement equipment"; 

Part 4: "Requirements for Quality of Service measurement equipment"; 

Part 5: "Definition of typical measurement profiles"; 

Part 6: "Post processing and statistical methods". 

Part 1 identifies QoS aspects for popular services in GSM and 3G networks. For each service chosen QoS indicators are 
listed. They are considered to be suitable for the quantitatively characterization of the dominant technical QoS aspects 
as experienced from the end-customer perspective. 

Part 2 defines QoS parameters and their computation for popular services in GSM and 3G networks. The technical QoS 
indicators, listed in part 1, are the basis for the parameter set chosen. The parameter definition is split into two parts: the 
abstract definition and the generic description of the measurement method with the respective trigger points. Only 
measurement methods not dependent on any infrastructure provided are described in the present document. The 
harmonized definitions given in the present document are considered as the prerequisites for comparison of QoS 
measurements and measurement results. 

Part 3 describes typical procedures used for QoS measurements over GSM, along with settings and parameters for such 
measurements. 

Part 4 defines the minimum requirements of QoS measurement equipment for GSM and 3G networks in the way that 
the values and trigger-points needed to compute the QoS parameter as defined in part 2 can be measured following the 
procedures defined in part 3. Test-equipment fulfilling the specified minimum requirements, will allow to perform the 
proposed measurements in a reliable and reproducible way. 

Part 5 specifies test profiles which are required to enable benchmarking of different GSM or 3G networks both within 
and outside national boundaries. It is necessary to have these profiles so that when a specific set of tests are carried out 
then customers are comparing "like for like" performance. 

Part 6 describes procedures to be used for statistical calculations in the field of QoS measurement of GSM and 3G 
networks using probing systems. 
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Introduction 



All the defined quality of service parameters and their computations are based on field measurements. That indicates 
that the measurements were made from customers point of view (full End-to-end perspective, taking into account the 
needs of testing). 

It is assumed that the end customer can handle his mobile and the services he wants to use (operability is not evaluated 
at this time). For the purpose of measurement it is assumed that: 

• the service is available and not barred for any reason; 

• routing is defined correctly without errors; and 

• the target subscriber equipment is ready to answer the call. 

Voice quality values measured should only be employed by calls ended successfully for statistical analysis. 

However, measured values from calls ended unsuccessfully (e.g. dropped) should be available for additional evaluations 
and therefore, must be stored. 

Further preconditions may apply when reasonable. 
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Scope 



The present document describes typical procedures used for QoS measurements on e.g. telephony services over GSM, 
along with settings and parameters for such measurements. 

Where possible existing ITU-T or ETSI definition are referenced. In some cases ITU-T or ETSI definitions do not exist 
or are considered as too generic, then a more service and mobile network specific definition is chosen. 



References 



The following documents contain provisions which, through reference in this text, constitute provisions of the present 
document. 

• References are either specific (identified by date of publication and/or edition number or version number) or 
non-specific. 

• For a specific reference, subsequent revisions do not apply. 

• For a non-specific reference, the latest version applies. 

Referenced documents which are not found to be publicly available in the expected location might be found at 
http://docbox.etsi.org/Reference . 
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ITU-T Recommendation P.56: "Objective measurement of active speech level". 

ETSI TS 102 250-1: "Speech Processing, Transmission and Quality Aspects (STQ); QoS aspects 
for popular services in GSM and 3G networks; Part 1 : Identification of Quality of Service 
aspects". 



Abbreviations 



For the purposes of the present document the following abbreviations apply: 

APN Access Point Name 

DNS Domain Name Server 

FTP File Transfer Protocol 

GGSN Gateway GPRS Support Node 

GPRS General Packet Radio Service 

HTTP Hyper Text Transfer Protocol 

IP Internet Protocol 

MO Mobile Originated 

MOS Mean Opinion Score 

MT Mobile Terminated 

MTU Maximum Transmission Unit 

PABX Private Automatic Branch eXchange 

PC Personal Computer 

PDU Packet Data Unit 

POP3 Post Office Protocol version 3 

QoS Quality of Service 

SMPT Simple Mail Transfer Protocol 

SMS Short Message Service 

SMSC Short Message Service Centre 

TCP/IP Transmission Control Protocol/Internet Protocol 

UDP User Datagram Protocol 

URL Uniform Ressource Locator 

WAP Wireless Application Protocol 
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Goal of measurement 



The goal of measurements described in the present document is to assess the network under test for its quality 
parameters as defined in TS 102 250-1 [2]. This is, to determine the network quality for the respective transactions from 
subscribers view. 



General aspects for all types of services 



5.1 Set-up and control 



Measurements should be conducted in a way that user behaviour is emulated, with a number of parameters under 
control of the measurement equipment. 

The test case design (configuration and user profile) - to the degree necessary to fully reproduce the test - shall be part 
of the report. 

It is assumed that for all types of services under test, a testcase consists of a number of single identical transactions. The 
measurement equipment and control must ensure that the starting conditions are the same for each transaction. This 
includes, among other things, that pause times are sufficiently long that the equipment is in a stable (idle) state again. 
The parameter "guard time" sets a minimum value for the pause between transactions. 

It is assumed that the measurement is performed by a mobile measurement unit; depending on type of measurement, 
other equipment connected to the fixed network is added. 



6 Speech measurements 

6.1 Measurement set-up and control 

The basic transaction for speech testing is equivalent to a single call to a counterpart extension. 

To ensure comparability and statistical validity of transactions, the following outer conditions need to be constant 
throughout a test case: 

• Call counterpart. This includes the type of equipment (dedicated unit, automatic reply with taped message 
etc.). It is assumed that the call partner is typically a fixed-network type extension to avoid uncertainties 
related to a second mobile connection. 

• Call timing including behaviour in case of dropped calls. For reasons of comparability in density, it is required 
that all call attempts which form part of the statistics must follow a constant time pattern. Additional call 
attempts must not affect the pattern of the statistically relevant call attempts (see figure 1). 
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Figure 2 shows the overall structure and set of parameters as an overview. 
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Figure 2 

The measurement equipment and control must ensure that the starting conditions are the same for each individual 
transaction. This includes, among other things, that pause times are sufficiently long that the corresponding call has 
been cleared down completely and that the equipment is in a stable (idle) state again. The parameter "guard time" sets a 
minimum value for the pause between calls. 

Clause 6.1.1 discusses the call transaction structure in more detail. 

6.1.1 Call Types 

The call type is either Mobile Originated (MO) or Mobile Terminated (MT). 

Basically, it is assumed that once a connection has been established, for further measurements it does not matter which 
side has triggered it. Therefore, the audio data flow parameter will not be logically linked to the call type. 



6.1 .2 Call transaction phases 



6.1.2.1 



Call set-up 



For call set-up assessment beyond QoS data acquisition, typically a state model driven by Layer 3 information 
combined with information from the call control engine is being used. This state model may also be used to determine 
timing information for each phase. 

A typical state model allows for a more detailed information about the outcome of a call attempt. The following terms 
are defined: 

• Access Failure: Network access failed. 

• Setup Failure: Network access was reached, but the call did not reach the state of an usable two-way 
end-to-end connection. This shall be verified by a procedure based on audio test transmissions within a given 
time window. If within this time window no audio connection can be verified, the setup attempt shall be 
considered to be failed and the call attempt be terminated. 

A call attempt which leads to an usable connection is termed an Setup Success. Call setup times shall, however, be 
determined from the logical event indicating a connection so the connection-verification method will not influence the 
measurement. 

This leads to the following decision tree for the outcome of a call (figure 3 includes the end-of-call cases). 
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Figure 3 

The following QoS objects are determined from the call set-up phase. 



6.1.2.1.1 



Service Accessibility 



This is a "try/success" counter type object. A try is recorded when a call attempt has been made. A success is recorded 
when a Setup Success is detected. 



6.1.2.1.2 



Setup Time 



This is a "time" type object. Start corresponds to the respective "try" trigger point of the Service Accessibility object 
(see clause 6.1.2.1.1). Stop trigger point is set by the event indicating logical connection. 



6.1.2.2 



Call connect 



The call connect phase starts when the connection has been established fully, defined by the success condition of call 
set-up. 

Audio flow direction is an inner parameter for a transaction. Basically, all combinations of uplink/downlink dynamics 
are possible: 

• Uplink only; 

• Downlink only; 

• Conversational (alternating uplink and downlink). This is the recommended standard testing mode. Other 
testing modes are considered to be used only for special purposes; 

• "Duplex" (uplink and downlink audio simultaneously). 

Only calls with a valid two-way end-to-end audio connection shall be considered for speech quality assessment (valid 
calls). To make sure an audio connection is valid, it is assumed that an appropriate kind of data analysis on audio flow 
is performed (see ITU-T Recommendation P. 56 [1]). 

A non-valid call shall be treated like a dropped call, with a modifier indicating this particular cause. 
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The following QoS objects are determined from the call connect phase. 

6.1.2.2.1 Speech quality 

For speech quality assessment, data is generated at the receiving end. For downlink speech, data storage is therefore 
straightforward; speech assessment data is simply included with other data items making up the result file. For uplink 
speech, at some point in time results have to be integrated with the rest of the data. 

For assessing speech quality, at least the following methods are possible: 

• Real-time assessment (streaming mode), where the speech quality assessment algorithm continuously outputs 
MOS data items; 

• "Offline" assessment, where speech is first recorded in some way and later being processed. 

Only speech quality data resulting from non-dropped calls shall be considered at all. 

Data processing must make sure that only such speech quality data is used which lies inside the "connection active" 
time window. 

6.1 .2.2.2 Call completion rate 

A call is considered to be completed when it is active until the measurement equipment actively ends it. 

A call which ends prematurely is termed a dropped call. 

A call is active only as long as both sides consider it to be active. A call is therefore considered to be dropped if either 
side detects a dropped call. 

The Call Completion Rate is computed as a try/success type object, where a try is recorded upon Setup Success, and a 
call success is counted when the measurement unit determines that the call has not dropped. 

6.1.2.3 Call clear-down 

The call clear-down phase is the transition between the connected state and the idle state. No QoS parameters referring 
to call clear-down have been defined. Typically, measurement equipment will, however, record Layer 3 sequences 
which can be used to determine specific information about clear-down. 

6.1.2.4 Pause 

This phase is intended to give the mobile and the network time to reach its neutral state again, so the next transaction 
starts having the same conditions as the one before. For current GSM equipment, a pause time of at least 15 seconds 
(guard time) is recommended. 

However, this duration may be adjusted to local conditions or special testing goals, but this must be reported. 

If the pause duration is too short, side effects may occur, resulting in all kinds of transient effects and distortions in 
measurement data. It should be made certain that all the QoS parameters to be measured are not affected by the pause 
time. 
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7 SMS measurements 

7.1 General aspects 

The basic transaction for SMS testing is equivalent to transmission of a single SMS. 

Typically, user SMS are being sent either from some fixed-network source to a mobile (SMS mobile terminated, 
SMS-MT), or from one mobile to another (SMS mobile originated, SMS MO, with regard to mobile-based testing). 
However, uplink SMS testing with a two-mobile setup may cause additional uncertainty due to coverage and signalling 
effects in the destination mobile. Basically, there are two recommended methods of testing: 

a) using a destination mobile in a fixed location with virtually 100 % transfer probability. Part of the 
measurement procedure should be cyclical reference checks to assure reception quality respectively to create 
baseline data; or 

b) using a fixed-network destination such as a Large Account SMS server which is accessed by suitable means 
from a fixed-network location. 

In all cases, testing methodology requires that measurement data are being collected in at least two locations and need 
to be integrated before the final QoS evaluation can be made. 

There are two modes form SMS transmission: 

a) text mode; 

b) PDU mode. 

Basically, these modes should be equivalent including options available. In practice, mode support are 
mobile-dependent. 

For each SMS, a "lifetime" can be set after which a SMS is deleted in the SMSC. While in PDU mode, this parameter is 
part of the parameter structure, in text mode it is set in the mobile, which may not be supported by all mobile types. 
This leads to the recommendation that PDU mode should be used. 

To ensure comparability and statistical validity of transactions, the following outer conditions need to be constant 
throughout a test case: 

• SMS mode; 

• Call counterpart; 

• SMSC being used 

• Call timing including behaviour in case of delivery failure to the network. 

For reasons of comparability in density, it is required that all SMS sending attempts which form part of the statistics 
must follow a constant time pattern. Additional SMS sending attempts e.g. in the case of failure of delivery to the 
network must not affect the pattern of the statistically relevant call attempts. 

SMS being sent from either side should contain information enabling validity and integrity checking during evaluation. 
The recommended minimum set includes: 

• Signature assuring that only test system SMS are being considered; 

• sequence number for SMS delivery timing. 

If required, a test SMS may contain additional data (padding) to obtain SMS with a length assumed to be typical for a 
particular network. 

Proposed: If required, more powerful means of ensuring that only SMS generated by the test system can be added. For 
example, the SMS may contain a code which is created from checksum content and a seed value delivered by the 
receiving side. This ensures that even if a static signature is duplicated by operational errors or malfunction of other test 
systems, the system will be robust against it. 
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Further for general design of SMS testing, communication between the mobile testing system and its stationary 
counterpart is required to ensure certain starting conditions: 

• Prior to actual testing, the SMS storage of the MS should be cleared to exclude SMS delivery failure due to 
memory shortage; 

• The stationary side should start sending SMS only after a "go" from the mobile side, ensuring that no transient 
effects occur. 

Since SMS is a store-and-forward service, SMS loss rate needs precise definition. From the user's point of view, there is 
expectation that a SMS is delivered within a certain time window. Even if technically an outstanding SMS may be 
delivered later, it should be considered lost anyway. This leads to the problem of over-aged SMS which may disturb 
subsequent measurements. Suitable means of eliminating such effects should be taken, e.g. setting a reasonable SMS 
lifetime, and using unique sequence numbering in a way to prevent aforementioned effects. 

7.2 Testing modes 

7.2.1 SMS-MT 

7.2.1.1 Event flow 

For this type of testing, SMS are being generated at a constant rate by the stationary side. 

All SMS received which have not been generated by the stationary side shall be ignored for quality assessment. 

For reception behaviour, two modes exist; it is mobile-dependent which modes are supported. 

a) On SMS reception, the mobile generates an indication message (CMTI message) on its data connection to the 
PC. The actual SMS can then be requested by a command. 

b) The mobile outputs the SMS directly on its data connection to the PC. 
From control and trigger point accuracy considerations, mode a) is preferred. 

7.2.1.2 Specific QoS Objects 

Network accessibility. 

SMS delivery time to network. 

7.2.2 SMS-MO 

7.2.2.1 Event Flow 

For this type of testing, SMS are being generated at a constant rate by the mobile side, plus eventual extra SMS in case 
of failure of delivery to the network. 

7.2.2.2 Specific QoS Objects 

SMS acquisition time (from paging to getting the SMS). 

7.3 Common QoS Objects 

These objects are being determined for MO and MT, but need to be treated "by direction" anyway. 

SMS end-to-end delivery time. 

SMS fail rate (lost SMS with respect to a given time window). 
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SMS duplicate rate. 



Probability that a SMS has more than zero bit errors (assuming the worst case, i.e. that even a single bit error makes the 
SMS useless for the subscriber). 

Probability that SMS are being received which have been sent by others (during our tests, we found that such effects 
exist; this surely needs some discussion). 



8 Data measurements 



In the following clauses, data measurements for internet-related services are described. While process of obtaining a 
connection is different between Circuit-switched and Packet-switched access, there are many similarities for the actual 
data transfer phase. 

8.1 Common aspects 
8.1.1 Target for data access 

The following categories of servers are distinguished, it is assumed that data service access is made to a server or entity 
within the general internet domain: 

• Third-party content in the public internet. We will term this type of counterpart A-Servers; 

• Accounts on internet servers which are under control of the testing system or under control of testing 
personnel (e.g. web domains assigned for testing purposes). This type of counterpart shall be termed 
B-Servers; 

• Special servers not reachable via public internet (e.g. in the GGSN domain), or equipped with additional 
instrumentation e.g. for IP-level tracing or non-internet capabilities (e.g. UDP testing). This type of 
counterpart shall be termed C-Servers. 

Access to a particular type of server will be termed using the same letter, e.g. A-access for access to an A-server. 

It is assumed that A-servers are generally outside the sphere of control of testing systems, in particular, no baseline 
information for load or other conditions is obtainable. In case of B-servers and C-servers, such control may exist, but 
will typically be limited in some way due to IP-security reasons, in particular if IP access occurs intra-network. 

Typically, when testing internet services, availability may depend on influences other than those under test, and 
performance will be affected by third-party traffic. Meaningful tests therefore shall contain appropriate measures to 
exclude such effects from QoS assessment, or make sure that all networks under test are affected the same way. It is 
assumed that such tests are performed by fixed network units. Suggested methods are: 

• Cyclical availability checks on target servers or domains; 

• Cyclical access-time tests. 

It is assumed that for services where login is required, accounts used by the test system are valid, give positive login, 
and are good for full access to all activities forming the test. 

NOTE: This covers read/write privileges and directory-access rights e.g. for FTP). 

User's point of view typically includes assumptions of a time the user is ready to wait before an action is considered to 
be failed. At the same time, IP service access typically goes with timeout windows at several levels., e.g. for in activity 
over a certain period of time. Test design must combine these two aspects to a reasonable, technically feasible set of 
parameters. 



ETSI 



1 4 ETSI TS 1 02 250-3 V1 .1 .1 (2003-1 0) 



8.1.2 Test Data Content 



When using web content (web sites or single pages) for testing, it must be taken into account that such content will 
typically change frequently (e.g. content of popular web portals) and therefore performance tests may give varying 
results over time. Test design must ensure that such effects are excluded from QoS assessment. Preferably, standardized 
and constant web content shall be used. 

The degree of control a testing system has further depends on the type of service. 

With E-Mail and FTP (assuming appropriate access privileges), data content can be determined exactly (e.g. by 
uploading files to be downloaded as part of subsequent tests. 

8.1 .3 Outer conditions to be kept constant for testing 

To ensure comparability and statistical validity of transactions, the following outer conditions need to be constant 
throughout a test case: 

DNS used (by network); 

APN and other initial settings; 

Target URL, or server IP address; 

Account being used (where appropriate); 

Access timing including behaviour in case of failure to obtain IP access. For reasons of comparability in 
density, it is required that all access attempts which form part of the statistics must follow a constant time 
pattern. Additional attempts e.g. in the case of network or service unavailability must not affect the pattern of 
the statistically relevant access attempts. 

Further for general design, the following service-dependent measures shall be taken: 

• FTP: For download tests, it must be assured that the file to be downloaded is actually available on the server. 
For upload tests, it must be assured that no storage-size limitations prevent successful upload. 

• E-Mail: Proper authentication must be assured. For download tests, it must be assured that the mails to form 
download traffic are actually present on the server. For upload tests, it must be assured that no storage-size 
limitations prevent successful upload. 

8.1 .4 Definition of transaction 

The basic transaction for data services testing consists of network access followed by a single data transaction such as 

• downloading or uploading a single FTP file of given size; 

• downloading a web site for a given URL response of given structure; 

• downloading or uploading a given number of e-mails of given size. 

It is understood that basic access procedures such as server login shall not be considered as part of a transaction. 
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8.1 .5 Assessing transactions 

From user's perspective, outcome of a transaction can either be: 

• Success: The intended transaction has been completed. 

• Session loss: Network or IP connection has been lost before completing the transaction. 

In case of session loss, the testing system shall indicate which of the possible principal causes occurred for the purpose 
of deciding of the network under test is to be blamed or not: 

• Loss of radio connection. 

• Loss of basic IP connection. If DNS access is still possible after a session loss, it is assumed that basic IP 
connection is still intact. 

• Loss of internet access: If access to another domain or server is still possible, it shall be assumed that basic 
internet. 

• Loss of connection to the server. 

8.2 Circuit switched 

A typical CSD data access consists of the following phases: 
| Connection set-up | Session access | DNS access | Domain access | Data transfer | 

Connection set-up: Setting up the basic network dial-up connection. Successful connection set-up results in a network 
connection usable for service access. 

Session access: Access and authentication procedure for basic internet access. Successful access is assumed when a 
temporary IP address good for data transactions has been assigned to the testing system. 

DNS access: Obtaining an IP address from a URL (web site name, server name). Successful DNS access is assumed 
when an IP address has been obtained from a given, valid URL. 

Domain access: This is the actual internet access as seen from the user's perspective. For testing purposes, a successful 
domain access can be assumed when communication with the target server has been proved. The respective action will 
depend on the service under test. 

Data transfer: In this phase, actual user-data flow occurs (file or mail upload or download). Depending on service, 
there may be several repetitions of domain access-data transfer phases. 



8.3 Packet switched (GPRS) 

A typical CSD data access consists of the following phases: 



| Network access | Session access | DNS access | Domain access | Data transfer 

Network access: Obtaining basic network access ability. In GPRS, this is the Attach to the network. 

Session access: Access and authentication procedure for basic internet access. Successful access is assumed when a 
useful PDP context (with IP address other than 0.0.0.0) has been assigned by the network. 

DNS access: Obtaining an IP address from a URL (web site name, server name). Successful DNS access is assumed 
when an IP address has been obtained from a given, valid URL. 

Domain access: This is the actual internet access as seen from the user's perspective. For testing purposes, a successful 
domain access can be assumed when communication with the target server has been proved. The respective action will 
depend on the service under test. 
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Data transfer: In this phase, actual user-data flow occurs (file or mail upload or download). Depending on service, 
there may be several repetitions of domain access-data transfer phases. 

Typically, in GPRS longer periods of inactivity can occur with the session still intact. On the other hand, it must be 
taken into account that a useful PDP context is indicated by the system, which is in fact not useful. Therefore, testing 
shall include cyclical "lifecheck" measures. A possible method are ICMP pings to the DNS, not if such pings are 
blocked by the network, DND accesses with dummy URL. Due to possible URL/IP address storage, it must be assured 
that actual DNS access takes place. 



9 Information to be logged for reproducibility of 

measurements 

The following information elements need to be logged in order to ensure reproducibility and comparability of tests. 

9.1 Speech measurements 

• Call counterpart type (e.g. public telephone network, PABX). 

9.2 SMS measurements 

• SMSC used. 

• Notification mode used for incoming SMS. 

9.3 Data measurements 

Operating system (type and version). 

MTU size. 

Logical location of server (public internet, GGSN, etc.) with respect to effects possibly created by other traffic. 

Maximum server throughput inbound and outbound, per session response per connection. 

Type of access (IP address vs. URL). 

DNS used. 

9.3.1 HTTP 



• 



Type of browser used (including version number/build). For test-system browsers, maximum number of 
parallel socket connections. 



• Reference web site used (structure, size of documents etc.) 

9.3.2 FTP 

• Type of FTP client used. 

• Protocol used (TCP/IP or UDP). 
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9.3.3 E-mail (SMPT/POP3) 



• Type of e-mail client used (including version number/build). For test-system clients, maximum number of 
parallel socket connections. 

• Authentication mode used. 

9.3.4 WAP 

• Gateway. 

• Type of access (text, binary connection-oriented, binary connectionless). 

9.3.5 UDP 

• Block size uplink/downlink. 

• Stream rate uplink/downlink. 
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